The International Olympic Committee made the decision after speaking with Japanese Prime Minister Shinzo Abe and local organizers.